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ABSTRACT 


The benefit of software cost estimation is universally recognized as one of the cor- 
nerstones of effective software project management and control. Despite the advances of 
computer-based estimation tools, their accuracy remains largely inadequate, and their 
utility among software development practitioners is limited. Consequently, the optimal 
estimation of software cost remains an elusive goal of most project managers. Central to 
this tssue is the nature of the data on completed software projects that are incorporated 
into the organization's database of historical project results. This information forms the 
basis for both future project estimation and ex-post-facto assessment of estimation mod- 
els. Actual project results are typically the data of choice for both the calibration and 
evaluation processes, despite the fact that these raw values disregard project inefficien- 
cies such as initial size underestimation. This thesis challenges the notion that historical 
project results represent the preferred and most reliable benchmarks for future estimation 
purposes. Computer-based simulation is used to test a proposed strategy which capital- 
izes ON an organization's learning experiences by neutralizing the cost excess caused by 
the initial undersizing, and that derives a posterior set of nurmulized effort and schedule 
estimation benchmarks. Analysts of the results indicates that normalization of the data 
leads to significantly improved project productivity. more optimal cost estimates. and 


provides the organization with increased potential for future cost savings 
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{. INTRODUCTION 


A. BACKGROUND 


The benefit of reliable software cost estimation is recognized as one of the comer- 
stones of effective software project management and control (Boehm, 1981, p.30). Nev- 
ertheless, accurate estimation of software development costs remains an elusive goal of 
most project managers, despite the proliferation of software engineering economic analy- 
sis techniques and the availability of computer-based software project management tools 
(Abdel-Hamid. 1990, p.71). 

Software development has traditionally been viewed as a discrete set of software de- 
velopment life cvcle (SDLC) phases, when in fact, research findings point to a dynamic 
environment characterized by continuous changes over time (Goddard Space Flight Cen- 
ter, 1990) Consequently, problems inherent with the estimation process itself, normally 
positioned at the beginning of the SDLC, have generally limited the utility of estimation 
tools based on this traditional view of software development. 

Without the benefit of full knowledge of a project's ultimate scope and definition at 
the time of initial cost estimation, an estimation model must possess the capability to re- 
spond to influencing factors which unfold as the project progresses through the SDLC. 
Abdel-Hamid states that "...estimation should be a continuous process enhanced through 
constant updates of feedback data collected from project monitoring and control activi- 


ties..." 


He argues that continuous estimation models must support the full range of 





estimation activities regularly encountered in the SDLC, adaptive (accommodate new or- 
ganizational realities), corrective (correct initial faulty assumptions) and perfective (post- 
mortems to perfect project statistics). In so doing, it 1s imperative that the model also 
possess the capability to capture management-system dynamics -- project managers’ reac- 
tions to real-world events as they unfold. (Abdel-Hamid, 1993. pp. 20-21) 

Despite the improvements realized with the introduction of genuine continuous esti- 
mation moacis, their accuracy remains largely inadequate. Central to this issue 1s the na- 
ture of the data on completed software projects that are incorporated into the 
organization's database of historical project results. This archived information subse- 
quently forms the basis for both future project estimation and ex-post-facto evaluation of 
software cost estimation models. Quite simply, this data is used to produce the organiza- 
tion's "best guess” of what a project of similar size and scope should require, in terms of 
development effort and schedule, if encountered in the future. In addition, it is these data 
values upon which estimation tool calibration, or fine-tuning to produce more accurate 
estimates which reflect the organization's unique software development environment, is 
based. 

Raw project values, which represent actual results, are the conventional “data of 
choice" for both the estimation and calibration processes. While raw data, indeed, reflect 
actual results, they may certainly not reflect optimum results, particularly in the case of a 
problematic project. Inefficiencies such as initial size underestimation, plague many, if 


not all software development projects, and are manifested in varying degrees of cost 


tv 





overruns and schedule slippages. As such, direct incorporation of raw values into the his- 
torical database tends to discount the impact of these inefficiencies on project results. In- 
stead, it merely archives this flawed information for future (mis)application, and 
perpetuates the cycle of inefficiency and imprecise estimation. 

In response, Abdel-Hamid has proposed a strategy which "...capitalizes on an organi- 
zation's learning experiences, by wringing out the cost excess caused by the initial under- 
sizing and that derives a posterior set of normalized cost and schedule estimation 


benchmarks." (Abdel-Hamid, 1993, p. 28) These normalized values are representative 


of a perfectly-sized software project, and consequently should provide the organization 


with a more efficient benchmark for future project estimation and planning, and in retro- 
spect, evaluating how well project resources were used. 
B. PURPOSE OF RESEARCH 

This research challenges the notion that raw historical values represent the preferred 
benchmark for calibrating software cost estimation models. Computer-based simulation 
is used to model the behavior of a number of synthetic project profiles to test the assump- 
tions of both the conventional and normalized strategies for software estimation model 
calibration. Various experimental conditions are imposed on subsequent experiments to 
compare project results and identify causal relationships in an effort to substantiate the 


research claims. 





C. THE RESEARCH QUESTION 


The primary research question of this thesis 1s to determine if there is long-term bene- 
fit in using normalized software project cost values vice raw historical data as the bench- 
mark for calibrating software estimation models. 

D. SCOPE OF RESEARCH 

The scope of this research includes the design, execution and analysis of a computer- 
simulated, multiple-project experiment, and comparing the results of two competing soft- 
ware estimation calibration strategies, in order to answer the research question. Its scope 
does not extend beyond the research laboratory, and there are no immediate plans for 
replicating this experiment in a real-world environment. 

E. THESIS ORGANIZATION 

Chapter I} offers a statement of the experiment’s objectives and a comprehensive de- 
scription of the experimentation tools, to include the COnstructive COst MOdel of Soft- 
ware Cost Estimation (COCOMO) and the System Dynamics (SD) Model of Software 
Project Development. In addition, Chapter I presents the experimental design, where the 
hypothetical projects, project profiles and influencing factors and assumptions are de- 


fined in detail. A key element of Chapter II is a discussion of the competing software es- 


timation model calibration strategies which form the basis of this research. Chapter III 


describes the experimental setting and related tasks, and elaborates on exercise organiza- 
tion, methodology and conduct. In addition, the dependent measures which represent key 


exercise metrics, are defined as they relate to analyzing and comparing exercise results. 





Chapter IV presents the results of the vanous experiments and offers insight and analysis 
of the research findings. Chapter V summanzes the findings of the previous chapters, 
discusses the implications of this study, and proposes related opportunities and directions 


for future research. 





il. METHOD AND PREPARATION 


A. EXPERIMENTAL OBJECTIVE 


This experiment will use a system dynamics model of software development to simu- 
late the development of a set of 30 projects in a software organization, conductec iV 
over an approximate 12-year period. The simulated results will be incorporated into an 
organizational data base and used as the basis for both subsequent project estimation and 
calibration of the estimation tool. Two scenarios will be evaluated: the conventional! 
method of calibration using raw historical data and an alternative calibration method us- 


ing "normalized" metrics. 


B. EXPERIMENTATION TOOLS 
1. Constructive Cost Model (COCOMO) 


The COnstructive COst MOdel, or COCOMO, was developed by Barry Boehm, and 
is a widely-accepted algonthmic model which is used to determine initial software 
development effort and schedule estimates. As a result of model refinement since its 
introduction, three model versions and three software development modes have evolved. 
The three versions include Basic, Intermediate and Detailed COCOMO, each of 
increasing detail and accuracy. Organic, Semidetached, and Embedded software 
development modes have been defined to accommodate the broad spectrum of project 


size, specificity, and risk encountered 1n the software development environment. 


Basic COCOMO is the simplest version of the model, and is effective for rough order 
of magnitude estimates of software cost. However, Boehm cautions, "... its accuracy 1s 
necessarily limited because of its lack of factors to account for differences in hardware 
constraints, personnel quality and experience, use of modem tools and techniques, and 
other project attributes known to have a significant influence on software costs...." 
(Boehm, 1981, p. 58) With Basic COCOMO, estimates of effort are generated using only 
a single predictor variable, namely the number of delivered source instructions (DSI) 
developed by the project. 

Intermediate COCOMO improves upon the Basic version by incorporating an 
additional 15 predictor variables, or cost driver attributes, which are carefully identified, 
weighted and introduced in order to offset much of the cost variation found in Basic 
COCOMO. The 15 cost drivers are subdivided into four categor.es: software product 
attributes, computer attributes, personnel attributes, and project attributes. Each cost 
driver has an associated effort multiplier which is applied to the nominal development 
effort to obtain a more accurate estimate. Boehm contends that the level of accuracy 
achieved with Intermediate COCOMO "... is representative of the current state of the art 
in software cost models." (Boehm, 1984, p. 16) 

Detailed COCOMO provides the highest level of estimation accuracy by providing 
even more detail as model input. This is accomplished by employing a three-level 


hierarchical decomposition of the software product whose cost is to be estimated. In 





addition, phase-sensitive effort multipliers are used to accurately reflect the effect of the 


cost drivers on the phase distribution of effort. (Boehm, 1981, pp. 347-348) 

The three COCOMO modes of software development were defined as a result of 
research findings suggesting that software products of the same size often require varying 
degrees of effort and development time. Consequently, each of the COCOMO software 
development mode's effort and schedule equations will yield significantly different cost 
estimates. Hence, precise identification of the applicable mode, by means of its 
distinguishing features, is critical in order to prevent estimation inaccuracies. 

The organic mode represents projects that are relatively small in size, developed by 
small software teams in a generally stable development environment. Expenence levels 
are high, while schedule and performance pressures are generally lower. 

The semidetached mode represents the middle ground between the organic and 
embedded modes. Flexibility of approach is a trademark of the semidetached mode, as 
intermediate levels of project characteristics and a blend of organic and embedded mode 
characteristics may be encountered in the same project. 

Finally, the embedded mode represents a project that must operate within tight 
constraints. Requirements and interface specifications are generally inflexible, and can 
dictate a considerable need for innovative architectures, algorithms or functionalities. 
(Boehm, 1981, p.81) 

In this series of experiments, the Basic COCOMO version will be utilized as the 


software estimation model. While Intermediate COCOMO estimates have proven clearly 





superior, the rudimentary nature of the Basic COCOMO (only size input - no cost driver 
attributes) facilitates evaluation of model charactenstics in conjunction with the SD 
simulator. Likewise, the organic software development mode complements the choice of 
Basic COCOMO, and assumes a stable baseline software development environment in 


which the experiments can be conducted. 
2. A Dynamic Simulation Model of Software Development 
Research has underscored the impracticalities of controlled experimentation in the 


software engineering field as being excessively costly and time-consuming (Myers, 
1978). Simulation modeling provides a flexible and ideal environment in which 
competing assumptions and conditions may be tested. Unlike real systems, the effects of 
variable manipulation on internal system interactions can be isolated and more carefully 
studied. Consequently, for purposes of this experiment, simulation modeling was chosen 
as the experimental method by which the research question would be answered. 

The System Dynamics (SD) Model of Software Project Development, by 
Abdel-Hamid and Madnick, is a comprehensive, highly-detailed, quantitative simulation 
model which captures management-system dynamics and provides a continuous 
simulation capability. Based on the feedback principles of system dynamics, the model 
focuses on four interconnected subsystems, which integrate managerial decision-making 
activities (e.g., scheduling, productivity, and staffing) with the physical production of the 


software product (e.g., design, coding, reviewing, and testing). The four subsystems are 


10 





human resource management, software production, controlling, and planning. 


(Abdel-Hamid, 1993, p. 24) 

The purpose of the SD simulator is to serve as a laboratory vehicle for conducting 
experimentation into the dynamics of software development. As such, it provides a 
much-needed means by which the managenal side of the software development process 
might be more carefully examined and, hopefully, better understood. By design, the 
model does not deliver point predictions, but rather seeks to provide a general 
understanding of the nature of the dynamic behavior of a project. An important 
functionality of the model is the ability to perform sensitivity analysis, or "what-if" 
experiments, in order to develop a more complete understanding of the interrelationships 
of software development variables and identification of causal relationships. 

The model has been designed for use on medium sized, organic type software 
projects (i.e., projects that are 10,000 to 250,000 lines of code and conducted in familiar, 
in-house development environments) (Stephan, 1992, p. 13). For a detailed discussion of 
the model's actual structure, formulation and validation, see Abdel-Hamid and Madnick 


(1989 and 1991). 


C. EXPERIMENTAL DESIGN 
1. Definition of Experimental Projects 


Five hypothetical software development projects, of varying representative sizes, 
were initially defined and serialized as projects one through five. Their size was 


established in terms of thousands of delivered source instructions (KDSI) to match both 


the COCOMO and SD simulator input parameters. Table | presents project senals and 


their respective sizes, which remain fixed throughout all experiments. 


Project Serial Actual Size (KDSI) 
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Table |. Experimental Projects and Sizes 
2. Underestimation of Project Size 


Boehm states, “The software undersizing problem is our most cnitical road block to 
accurate software cost estimation.” He cites three main reasons for this perplexing 
phenomenon. First, people's optimistic and accommodating nature drive them to say 
what others want to hear. High estimates are fuel for confrontation, whereas everyone is 
happy with small, easy software. The second reason involves incomplete recall of the 
large amount of support software that must be developed as part of a project -- there ts 
generally a stronger recollection of the size and effort required for the much smaller, but 
more visible, operational software. The third reason is related to the incomplete recall 
issue. Unfamiliarity with the full scope of the software project causes people to overlook 
the more obscure software products (and obscure portions of each product) which need to 
be developed. There are no quick fixes to the pervasive undersizing problem other than 
to understand the sources of the problem, and apply that understanding to software sizing 


activities. (Boehm, 1981, pp. 320-323} 


A study of the impact of undersizing on software estimation forms the focus of much 
of this experiment. Consequently, underestimation levels, expressed as a percentage of 
actual project size, are applied to the individual project serials in accordance with the 


experimental project profile, which is defined in a subsequent section of this report. 


Underestimation levels are defined and presented in Table 2. 


Table 2. Project Size Underestimation Levels 
Undersizing has a direct effect on both the software cost mode) (COCOMO) and the 
simulation model (SD simulator) results. Quite simply, a too-small sizing estimate 
invariably results in a too-small cost estimate. For example, a 50 KDSI project, 
undersized by 20 percent, results in a Basic COCOMO estimation identical to that of an 


accurately-sized 40 KDSI project. 
3. Development of Project Profiles 


The experiment seeks to model and analyze the software development activities of a 
hypothetical organization over time. In developing a project profile for the organization, 
particular attention was paid to a number of conditions within the organization thai 
would accomplish exercise objectives, while maintaining a reasonable degree of realism 


with respect to the functioning of an actual software development organization. 
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a. Project Teams 


Five hypothetical software development teams are constructively assembled. As 
teams, they will be assigned to one of the project serials -- one team for each project 
serial. There was no consideration given to team make-up in assembling the teams. 
Although disregard for the effects of personnel attributes on team performance represents 
an exercise artificiality, the assumption of essentially "homogeneous" project teams 


facilitates unbiased interpretation of the exercise results. 
b. Project Cycles 


In order to investigate the long-term impact of calibration strategies on software 
cost estimation, follow-on projects to the five project serials already defined is required. 
Consequently, the concept of a project cycle is introduced. A project cycle is defined as 
that period of time required for each of the five individual project serials to be 
completed. The first iteration of this scheme is referred to as “Project Cycle One", 
whereas subsequent iterations are labeled "Project Cycle Two", “Project Cycle Three", 
etc. For purposes of this experiment, organizational software development activities will 
span six project cycles. 

c. Initial Project Team Assignments 


With teams assembled, and projects and project cycles defined, the next step is to 
determine a strategy for project assignment. Here the assumption is that all five software 
development teams will commence work on the five project serials concurrently, at time 


zero. For simplicity, and to provide a convenient project profile starting point, 


assignment of projects in project cvcle one matches team one with project one, team two 


with project two, etc.. Table 3 outlines cycle one project assignments. 


Project Cycle One 





Table 3. Cycle One: Team and Project Assignments 
ad. Allocation of Undersizing Factors 


In order to examine the effects of undersizing on projects of varying size, the 
previously -defined size underestimation levels (Table 2) must be allocated in a random 
manner across all projects. For project cycle one, this was accomplished by using a table 
of nuinbers generated by a random process. Table 4 is such a table and is used in the 
experiment By arbitrarily selecting the intersection of any row and column as the 
starting point, a list of five numbers is systematically drawn by moving either to the left 
or right. or upward or downward from this starting point until one of the underestimation 
level values 1s encountered. This number is recorded in the list, and the movement 
continues until a second number within the allowable range (one through five) is 
encountered. After this second value is recorded in the list, the process repeats three 


more times until the randomized list of five numbers is complete. For example. 


underestimation levels are allocated for project cycle one by choosing row 5, column 13 
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Table 4. Table of Random Numbers. After Ref. (Roscoe, 1975, p. 410) 
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(Table 4) as the starting point and moving across the row to the right. The following 
randomized list is generated: 4 - 2 - 3-5-1. These numerical values, corresponding to 


underestimation levels, are allocated to cycle one projects as shown in Table 5. 


Project Cycle One 


Undersizing Level 





Table 5. Cycle One: Projects and Undersizing Levels 

For project cycles two through six, undersizing levels are allocated in accordance 
with the Latin Square Design (Daniel and Terrell, 1975, pp. 209-215). Once the 
cycle-one undersizing levels are determined and allocated to the five project serials in 
ascending project-size order, Latin Square imposes a one-position downward shift of row 
values to produce the undersizing allocation for cycle two. The procedure is repeated 
through the six project cycles, which results in cycle-six undersizing levels identical to 
those in cycle one. Table 6 presents the undersizing allocation for all projects across all 
project cycles. This allocation plan is fixed, and is used for ail experiments where 


software size underestimation is assumed. 














Project Cycle 


projects] xosi {1 [2 [ 3 [ # Ts [ @ | 


Underestimation Level 


Table 6. Project Undersizing Allocation 


e. Project Team Assignments in Cycles Two through Six 


In developing the project profile, it was decided that when a project team 
completed their assigned project in cycle one, they would immediately be assigned a new 
project and commence work in cycle two. That is, the team that finishes their cycle-one 
project first, is assigned the first available project in cycle two. The second team to 
finish cycle one gets the next available project in cycle two, and so on, until all five 
teams "arrive" in project cycle two. Subsequent project assignments are determined in 
the same manner through project cycle five. 

The sequence of next-available projects for project cycles two through five are 
randomly assigned. Their project assignment orders are determined by employing the 
same randomization techniques described in the previous section, but with different 
starting coordinates and directions of movement for generating the randomized list for 


each cycle. 








To facilitate comparative analysis of results with cycle one projects, cycle six 
team assignments replicate their initial project assignments. Table 7 defines the 


next-available project scheme for all six project cycles. 


Order of 
Project 
Completion 
in Present 
Cycle 





Table 7. Next-Available Project Schedule 
SJ Finalized Experimental Project Profile 


The final project profile, which incorporates next-available project assignments 
and their respective undersizing levels, is presented in Table 8. All experiments follow 
this project-order and undersizing arrangement (when applicable). While project team 
assignments in other than the initial proyect cycle may vary under different exercise 
scenarios, depending on calculated total development schedule values, the follow-on 
project order and underestimation levels of Table 8 remain fixed in all cases. Figure | 
displays a representative Total Development Schedule for all five project teams over six 


project cycles, applying the experimental project profile. 
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Total Development Schedule 
(By Project Development Team and Project Cycle) 
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Figure 1. Total Development Schedule 





4. Learning 


The effects of "learning" on software estimation and productivity are an important 
element of this research. It is reasonable to assume that the effect of experience and 
increases in project familiarity should be reflected in higher productivity. In an attempt 
to model the rate of learning improvement, a plan involving the incremental increase of a 
related SD simulator input variable was developed. 

in the SD model, nominal productivity is defined as one task per man-dav. A task is 
any arbitrary unit by which a software project may be measured (Abdel-Hamid and 
Madnick. 1991. p. 80). In our experimentation vehicle, a “task” is defined in terms of a 
discrete number of Delivered Source Instructions, hence the SD input parameter 
Deliwered Source Instructions per Task (DSIPTK). | Consequently, an appropriate 
increase in [YSIPTK over the nominal simulator value as projects are developed, can 
effectively model the ‘learning curve' effect we are searching for. 

For purposes of this experiment, we assume that “learning” is reflected in a 
10-percent annual increase in DSIPTK. While total project development schedules 
obviously vary, an 18 to 24-month timeframe represents a reasonable estimate of 
duration for the hypothetical projects as defined. Consequently, a 20-percent increase in 
DSIPTK was applied to each project cycle beginning with project cvcle two. This value 
is consistent with research findings and industry experiences (Aron, 1976). Hence, the 


learning scenario is defined as an incremental increase of DSIPTK from 100 percent of 


tQ 
ta 





nominal value to 200 percent of the nominal SD simulator value over the six project 


cycles. Table 9 demonstrates how the learning scenario was applied. 


DSIPTK: Percent of 


Table 9. Leaming Scenario 
5. Conventional COCOMO Calibration Strategy 


“Calibration” is one method by which an organization may tailor a software 
cost-estimation tool to more accurately reflect its unique software development 
experiences. Boehm asserts that calibration of COCOMO may be necessary, for various 


reasons, to provide an organization with the best estimation accuracy "fit". He offers a 


technique for calibrating the constant term in the COCOMO nominal effort equation, and 


this procedure will be replicated as part of the experiment, and throughout the thesis will 
be referred to as the "conventional" calibration strategy. 

Having selected the Basic COCOMO model and the organic mode as the most 
appropriate software development mode for our hypothetical organization, the calibration 
methodology is straightforward. Table 10 presents the Basic COCOMO effort and 


schedule equations for the organic mode. A few terms require definition in 








understanding these equations. Under the /:/fort column, "MM" refers to the number of 


man-months estimated for the software development phase. One man-month is equal to 
152 hours of working time. Under Schedule, "“TDEV" is the number of estimated 


months for software development. 







Table 10. Basic COCOMO Effort and Schedule Equations (Organic Mode) 





The constant term in the effort equation above (2.4) is the value which is calibrated. 
Because of the absence of cost driver attributes in Basic COCOMO, the optimal 


coefficient may be calculated using the following equation: 
n 
C en D1 MM,(actual)*Q, 
Saas ac 
Lisl (Q;) 


In the above equation, MM (actual) is the actual development effort of the software 


(2.1) 


project. In our experiment, this value is generated by the SD simulator, based on input 
values which include the Basic COCOMO effort and schedule estimates. The variable Q. 
for organic mode re-calibration, is defined as the actual size of the project (KDSI(actua!)) 
raised to the power 1.05. Having determined these values, the calibration process 
continues by multiplying MM (actual) times Q, for each project. The summation of this 
product is determined for the number of projects being factored in to the re-calibration 
(n). This value forms the numerator of the re-calibration equation. The denominator is 
calculated by first squaring each Q, value, then summing these values. The resultant 


coefficient represents the derived optimal constant term and replaces the organic 


COCOMO coefficient value of 2.4 for estimation of the next senes of (n) projects. 


Chapter [V provides additional clarification of the calibration methodology using 


exercise data. 
6. Alternative "Normalization" Calibration Strategy 


Boehm commented on a comparative analysis of software cost models, that "...Not 
too surprisingly, the best results were generally obtained using models with calibration 
coefficients against data sets with few points..." (Boehm, 1984, p. 18). A similar 
analysis of the validity of the assumptions upon which calibration strategies are based, 
and their impact on software estimation model performance has received considerably 
less attention. 

Basic COCOMO embraces the assumption that historical project results represent the 
preferred and most reliable benchmarks for future estimation purposes. This experiment 
challenges that notion, and seeks to validate the work of Abdel-Hamid by using the SD 
model as an experimental vehicle to demonstrate why this assumption is flawed 
(Abdel-Hamid, 1990, p. 79). 

Using data from a real software project conducted by NASA, Abdel-Hamid 
conducted two experiments as part of SD model validation. The first experiment 
investigated one of two fundamental assumptions upon which conventional calibration 
strategies are based. That is, a project's final results are independent of its initial 
estimation values. His research findings indicate that different estimates do, indeed, 


create different projects. He reported that initial project effort and schedule estimates 





significantly influence work force level decisions, productivity, work intensity, and 
communication and training overheads. Clearly, acceptance of these findings leads to 
rejection of the convention that actual project results provide the best information for 
future estimation activities. 

Abdel-Hamid's second experiment sought to further refute the notion that raw 
historical project values should be the “data of choice" for both the calibration and 
ex-post-facto evaluation of estimation tools. Again, using the NASA data, he reported 
how the initial 35-percent size underestimation lead to a corresponding underestimate of 
project effort and duration. He observed how learning, in the form of increased project 
familiarity and experience, lead to the discovery of overlooked tasks, which in tum 
resulted in a dramatic "staff explosion" late in the development cycle, in order to meet a 
rigid deadline. At this point, the representativeness of NASA's actual project cost as the 
basis for future effort estimation becomes suspect due to the problematic nature of the 
project. A new project of similar size and scope, but more accurately sized at the outset, 
and consequently more effectively staffed, should result in project costs somewhat less 
than the actual results of NASA's troublesome effort. 

In his work, Abdel-Hamid outlines a "normalization" strategy for eliminating 
inefficiencies due to initial project undersizing which incorporates the capabilities of the 
SD simulator. Much of this research work is aimed at examining and testing this strategy 


against the conventional calibration strategy under a variety of conditions and scenarios. 


In theory, the normalization strategy seeks to determine the extent of man-day 
excesses, and adjust the archived calibration/estimation values accordingly. Figure 2 


diagrams both the current calibration practice and the proposed normalization strategy. 


RAW HISTORICAL CALIBRATION / 
RESULTS ESTIMATION 


RAW HISTORICAL “NORMALIZATION NORMALIZED CALIBRATION / 
RESULT TS ENGINE VALUES ESTIMATION 


(B) PROPOSED NORMALIZATION STRATEGY 





Figure 2. (a) Current Practice: (b) Proposed Normalization Strategy 

To determine the normalized cost value, a project must be re-simulated with no 
undersizing Optimization of cost savings is determined by repeated simulations in 
which actual project size and schedule inputs are fixed, while effort inputs are 
systematically reduced until further input reductions begin to yield higher cost outputs. 

The input and output values generated during a typical normalization process are 
presented in Table 1t. Repeated simulations in which actual project effort (MM(est)) is 
systematically reduced with each simulation, yields a series of actual costs (MM(act)). 
The shaded cell in Table 11 is the lowest numerical result generated by the SD simulator 


under all input conditions. This represents the project's "normalized" man-month value 


and reflects the optimum cost savings achievable in a perfectly-sized project. The 
estimated versus actual cost values of Table 11 are graphically represented in Figure 3 to 


further illustrate the normalization process 
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Table 11. Normalization Values 
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Figure 3. The Normalization Process 


I. CONDUCTING THE EXPERIMENT 


A. EXPERIMENTAL SETTING 


All experiments involve extensive simulation modeling and cos: estimation calcula- 
tions. In addition, archiving requirements for a significant volume of generated data 1s 
necessary, as well as relational processing capabilities to conduct comparative analysis of 
the findings. These requirements were satisfied, and the experimental tasks successfully 
accomplished on an [BM-compatible 486-DX2/66 personal computer (PC). 

The System Dynamics (SD) simulator runs in the MS-DOS environment, however the 
PC was configured to run the application in a window of Microsoft Windows 3.1, to fa- 
cilitate transfer of information. User interface is via the keyboard. Figure 4 is the 
"changes" screen, where input parameters are entered to examine the various exercise 
scenarios. Of note, the fields routinely used in experiment simulations are found on this 
screen such as DS/P7K and UNDEST (first column), 7O7MM (second column), and 
TDEV] (third column). A tailored report is also generated for each completed simula- 
tion, and provides not only a convenient presentation of simulation results, but also dis- 
plays initial input parameters to permit easy verification of data entry. A copy of one 
such report ts presented in Figure 5. 

An electronic spreadsheet, specifically Lotus 1-2-3, release 4.1 for Windows, was cho- 


sen as the appropriate application for manag?iig and presenting the experimental data. It 


offers advanced spreadsheet, charting, drawing, scenario and database features which 
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Figure 5. Tailored Simulation Report 
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were extremely valuable tools in conducting, analyzing, documenting and presenting the 
results of the experiment. 
B. RELATED EXPERIMENTAL TASKS 

With a clear statement of the experimental objective, appropriate choice of expen- 
mentation vehicles, and a valid experiment design, several administrative tasks remain 
to facilitate conducting the experiment and handling the data. Important to this pre- 
execution phase is the development of a number of worksheet templates in Lotus 1-2-3. 


The "calculations worksheets" are of particular value -- project profile data and simulated 


project cost data are directly entered here. Incorporated within the calculations work- 


sheets are numeric cell formulas and interrelationships such that upon appropriate entry 
of project data, key dependent values are automatically calculated. Figure 6 is an exam- 
ple of a calculations worksheet. A detailed explanation of the calculations worksheet's 
operation is presented with the research findings in Chapter IV. 

In addition, a number of tailored spreadsheet tables were developed to archive, per- 
form comparative analysis on, and display the collected data in a consolidated, readable 
format. Appendix B is an example of this type of tailored spreadsheet table. 

C. DEPENDENT MEASURES 

Answering the research question requires capturing key simulation and computational 

data on project performance and productivity. These values are absolutely essential to 


meaningful analysis and interpretation of the research findings. Each of these values is 
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Figure 6. Research Calculations Worksheet (Lotus 1-2-3) 





described below; parenthetical text following each heading reflects the abbreviation used 
for this value throughout the thesis: 

1. Actual Project Effort (MM(act)) 

Actual Project Effort is one of the dependent variables generated by the SD 
simulator, and represents the number of actual man-months required for the software 
development phase of each individual project. 

2. Actual Project Schedule (TDEV(act)) 

This value is also a dependent vanable generated by the SD simulator, and represents 
the actual number of months required for completion of the software development phase 
of each individual project. 

3. Actual Project Productivity (Productivity) 


Actual Project Productivity is an important metric by which competing calibration 


strategies are compared and evaluated. It is calculated by dividing the actual project size 


(KDSI(act)) by the actual project effort (MM(act)). This value is calculated ex-post-facto 
for each individual project. It is expressed as a decimal value, and there is an inverse 


relationship between actual project effort and actual project productivity. 
4. Composite Cycle Productivity (Comp Prod ) 


Composite cycle productivity is a deterministic value which reflects the combined 
productivity of all five projects as defined in a particular project cycle. It 1s calculated by 
dividing the total actual size of all projects in the cycle (summation of KDSI(act)), by the 
total actual effort of all projects (summation of MM (actual)). Since the total actual size 


of all projects in each cycle is fixed (300 KDSI), composite productivity is driven by the 





value of total project effort - the lower the total effort, the higher the composite 


productivity. 
5. Average Staff (Avg Staff) 


This value represents the average staffing level for each project. The accurate 
projection of required staff levels is a critical function in software development. 
Average Staff is calculated in COCOMO by dividing the actual project effort (MM(act)) 


by the actual project schedule (TDEV(act)). 
6. Normalized Project Effort (MM(norm)) 


Normalized Project Effort is the value resulting from the application of the 
normalization process, described in detail in Chapter II, to Actual Project Effort 
(MM(act)). Its value represents an optimal achievable level of project effort and forms 
the basis for calculation of the COCOMO Calibration Coefficient in the alternative 


calibration strategy which is examined in this experiment. 
7. COCOMO Calibration Coefficient (Coefficient) 


“Calibration” is one method by which an organization may tailor a software cost 
estimation tool to more accurately reflect its unique software development experiences. 
"Coefficient" refers to the constant term in the COCOMO nominal effort equation, and 
its calculated value is critical to subsequent model estimation accuracy. The central 
issue in the evaluation of the conventional versus the alternate (normalized) calibration 
Strategies involves the appropriateness of the independent variable upon which the 


coefficient calculation is based. In the conventional calibration strategy, it is based on 


36 


actual project effort (MM(act)), while the normalized calibration strategy bases its 
computation on normalized project effort (MM(norm)). 
D. ORGANIZING THE EXPERIMENT 

The experiment is conducted in four phases. Presented in this section of the report are 
the research objectives of the various experiments, an explanation of how each phase is 
organized, and a general explanation of the exercise "flow". Detailed process definitions 


are presented along with the experimental results and analyses in Chapter IV. 
1. Phase One 


The objective of this phase is to compare the simulated project cost results obtained 
by applying the conventional software estimation tool calibration strategy, against a 
similar set of cost values obtained by applying the normalized calibration strategy. Both 
learning and undersizing are assumed in this scenario. The project profile determines the 
project-set order and undersizing allocation for each of the six project cycles. The SD 
simulator and COCOMO equations are used to both replicate the conventional 
calibration strategy and test the alternative normalization strategy. Key computational 
values (Dependent Measures) are captured, and a comparative analysis of the two 
calibration strategies is offered. The data set collected in Phase One constitutes the "base 


case" results, against which all other scenarios are tested. 
2. Phase Two 


In Phases Two through Four, the experiment is structured to perform sensitivity 
analysis on the base case results. Different assumptions and environmental factors are 


examined by using the SD simulator's ability to change one input variable while holding 
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all others constant. In each scenario, particular attention is paid to the effects of 
"normalization", vis-a-vis the conventional calibration strategy, on the expenmental 
results. 

The objective of Phase Two 1s to examine the effects of size underestimation on base 


case results. A new case is developed where learning is assumed, but no_ size 





underestimation. Simulated results for the same project set are calculated, applying both 
the conventional and normalized calibration strategies, and compared with base case 


findings. All other conditions are identical to those in Phase One. 
3. Phase Three 


The objective of Phase Three is to examine the effects of learning on base case 
results. A new case is developed where undersizing is assumed, but no learning. 
Simulated results for the same project set are calculated, applying both the conventional 
and normalized calibration strategies, and cor.:pared with base case findings. All other 


conditions are identical to those in Phases One and Two. 
4. Phase Four 


The objective of Phase Four is to examine the impact of overestimation and 
underestimation of productivity on project-set results. In this scenario, we again assume 
undersizing and no learning, as in the previous experiment. However, this experiment 
explores the effect of misrepresenting productivity as a function of how the level of effort 
associated with the accomplishment of a software development "task" is defined within 


the organization. 
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Central to the productivity overestimation/underestimation question is the notion of 
"variable task definition." Disparate definitions of the effort required to accomplish a 


software task may account for situations where various software development organiza- 


tions require different levels of development effort to design and code projects of similar 


size and scope. In projects where the number of delivered source instructions is identical 
in each organization, the value of "task" becomes the determinant with regard to measur- 
ing effort, and hence, productivity. First, this experiment re-simulates the project set and 
examines the impact of underestimating productivity by a factor of 75 percent of the 
nominal case. Next, the project set is re-simulated, this time overestimating productivity 
by a factor of 125 percent of the nominal case. The results are compared to Phase Three, 
which models the nominal case in this scenario (undersizing, no learning, and "perfectly- 


represented” productivity). 
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IV. EXPERIMENTAL RESULTS AND ANALYSIS 


A. INTRODUCTION 


The SD simulation model generated raw data on the actual cost and schedule for each 
simulated project. The manner in which these values are applied in calibrating the CO- 
COMO software estimation tool, and its impact on productivity and cost savings under a 
series of conditions are the central foc::s of this analysis. As such, there are four princi- 
pal areas of investigation. First, the replication of a conventional software estimation 
tool calibration strategy using raw cost data and assuming both learning and undersizing, 
is compared with an alternative calibration strategy using normalized cost data under the 
same assumptio.is. Ne * ‘ase-case results of phase one are compared with simulated 
results of a new case assuming learning but no undersizing. The third area of investiga- 
tion is a comparison of the base-case results with a new case in which there is undersiz- 
ing, but without learning effects. Finally, the impact of both underestimation and 
overestimation of productivity on the results obtained in the scenario with undersizing 


and without learning is examined. 
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B. CONVENTIONAL VS. ALTERNATIVE CALIBRATION STRATEGIES WITH 
LEARNING AND UNDERSIZING (BASE CASE) 


1. Assumptions 
a. Underestimation of Project Size 
The Basic COCOMO schedule estimation mode] requires as its single input, a 
user-provided estimate of the project's size in thousands of delivered source instructions 
(KDSI). Consequently, an inaccurate size estimate input will result in a similarly impre- 
cise schedule estimation output. The inclination toward project size underestimation is 
not uncommon throughout the software industry (Boehm, 1981, p. 320). For purposes of 


this experiment, size underestimation, when applied, is represented as a percentage of 


actual project size. Undersizing is assumed to range from 10 percent to 50 percent, in 


10-percent increments, and is applied to individua! project serials in accordance with the 
project/cycle profiles presented in Chapter II. The undersizing percentages, expressed in 
decimal notation, are subsequently applied as the SD simulator input parameter 
UNDEST. 

b. The Effects of "Learning" on Software Estimation and Productivity 

By "learning" we mean increases in productivity. This learning happens as an or- 
ganization gains experience in developing its type of software and as it incorporates new 
software development tools. As discussed in Chapter II, we assume that “learning” is re- 
flected in a 10-percent annual increase in the SD simulator input parameter Delivered 


Source Instructions per Task (DSIPTK). Consequently, with project schedules generally 








approaching two years’ duration, a 20-percent increase in DSIPTK was applied to each 


project cycle beginning with project cycle two. Therefore, the learning scenario is de- 
fined as an incremental increase of DSIPTK from 100 percent to 200 percent of the 
nominal value over the six project cycles. 

2. Conventional Calibration Strategy 

Five synthetic project serials were simulated over six organizational project cycles, 
for a total of 30 simulations. Key computational values, as defined in Chapter III, were 
calculated and tracked throughout the experiment. They include Actual Project Effort 
(MM(actual)), Actual Project Schedule (TDEV(act)), COCOMO Calibration Coefficient 
(Coefficient), Actual Project Productivity (Productivity), Composite Cycle Productivity 
(Comp Prod), and Average Number of Staff Required (Avg.Staff). Appendix A presents 
all calculations and data used to generate these key values, which are further consoli- 
dated and summarized in Table 12. 

The methodology for determining actual simulated values will be described as the 
process unfolds in Appendix A. In the following discussion, descriptive abbreviations in 
parenthesis correspond to column labels in Appendix A. For each project serial (Proj Se- 
nal), a learning value (DSIPTK (%)) is assigned. A project size estimate (KDS\I(est)) is 
determined by multiplying the actual project size (KDSI(act)) times the size 
underestimation percentage (Under (%)). Using this project size estimate (KDS\(est)) as 


the input variable to the organic COCOMO formula, the estimated project effort 
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Table 12. Conventional Calibration Strategy 
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(MM(est)) and estimated project schedule (TDEV(est)) are determined. All required in- 


put parameters for the project simulation have now been calculated. They are, 
KDSi(act), DSIPTK (%) -- expressed as a numerical value based on the nominal simula- 
tor value of 60, Under (%) — expressed as a decimal value, MM(est), and TDEV(est). 
Next, the SD simulator generates the actual effort (MM(act)) and actual schedule 
(TDEV(act)) values. 

The second series of calculations presented in each project cycle in Appendix A, 
uses the simulated actual effort and schedule values of each of the five project serials to 
determine the COCOMO calibration coefficient (Coefficient) which will be applied to all 
projects in the subsequent project cycle. Coefficient calculation is based on a series of 
well-defined computations as described in Chapter II. In the case of project cycle one, 
the Coefficient of 2.56 reflects an upward adjustment from the organic COCOMO value 
of 2.4. If this "conventional" calibration strategy is effective, this higher value, when ap- 
plied to project cycle two size estimations, should produce more accurate effort and 
schedule estimates. Figure 7 shows the movement of the COCOMO calibration coeffi- 
cient over the six project cycles under the conventional calibration strategy. 

In addition, actual project productivity (Productivity) and composite cycle productiv- 
ity (Comp Prod) are also determined in Appendix A. Actual project productivity (Pro- 
ductivity) is defined as the actual size of the project (KDSI(act)) divided by the actual 
cost of the project (MM(actual)). Results of the experiment are displayed in Figure 8, 


and reflect individual preject productivities between .27 and .43. 
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Figure 7. COCOMO Calibration Coefficient: Conventional Calibration Strategy - Base Case 
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Figure 8. Productivity: Conventional Calibration Strategy - Base Case 





Composite cycle productivity is defined as the total actual size of all projects in the 


cycle (CXKDSI (act)), divided by the total actual effort of all projects (MM (act)). In 
the conventional calibration scenario, overall composite productivity of the software de- 


velopment organization through the six project cycles improved from .317 to .411 (29.65 
percent). Figure 9 captures this upward movement of composite productivity. 

3. Alternative Calibration Strategy 

The methodology employed in applying the alternative calibration strategy is identi- 
cal to the conventional strategy described in the previous section, with one important ex- 
ception. As described in Chapter 1, upon determination of actual cost and schedule 
values using conventional COCOMO techniques, the projects are re-simulated with ac- 
tual size and actual schedule inputs fixed. Cost estimates are gradually reduced from the 
actual simulated value until the optimum savings, or “normalized” cost value, is 
achieved. Appendix B provides all data on the normalization process for each of the five 
project serials over the six project cycles. Shaded cells in the MM(act) column represent 
the optimum or "normalized" value for that particular project. This value, referred to as 
MM(norm), is incorporated in the organizational data base and is used to calculate the 
new COCOMO calibration coefficient. Appendix C presents all calculations and data as- 
sociated with the calibration of COCOMO using normalized data. Note its similarities 
with Appendix A. However, in the second series of calculations for each project cycle, 
the normalized effort (MM{(norm)) is a new column entry. Its value was computed as 


part of the normalization process and transferred directly from the shaded cells in 
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Figure 9. Composite Productivity: Conventional Calibration Strategy - Base Case 











Appendix B. It is this value, MM(norm), which generates the new COCOMO coeffi- 
cient, and not the actual effort cost value (MM(act)), as in the conventional calibration 
strategy. 

It is important to note that normalization of the effort cost data has no direct impact 
on project productivity or composite cycle productivity, as actual effort costs continue to 
be used in computing these values. Normalization is primarily a process by which the in- 
efficiencies which have plagued a problematic software development project can be 
eliminated. In so doing, it is possible for an organization to optimize the accuracy and 
representativeness of archived data for future estimation of similar projects. 

A by-product of the nommalization process, however, is improved productivity. In 
theory, normalization provides the organization with more optimal calibration coeffi- 
cients which should lead to more optimal estimations. As inefficiencies are eliminated in 
project estimation, simulations produce projects with lower actual costs, which in turn, 
lead to improved productivity. These notions are borne out in the experimental findings 
summarized in Figure 10 and Table 13 - a comparison of the previously-determined raw 
historical data with the normalized data recorded upon re-simulation of the identical pro- 
ject set. Improvement percentages for normalized data versus raw data are calculated in 
Table 13 for actual cost, productivity, and composite productivity values. Note that be- 
ginning with project cycle two (when the normalization process first produces a unique 


calibration coefficient), improvement is noted across all entries. While improvements 
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Figure 10. Composite Productivity - Base Case 
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Table 13. Comparison of Conventional and Normalized Calibration Strategies - Base Case 
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are noted in productivity values associated with both raw and normalized data, the more 
dramatic results achieved through data normalization is apparent. 

Of particular significance is the improvement in composite cycle productivity evident 
within both the raw and nonnalized data sets themselves. Over the course of the six pro- 
ject cycles, composite productivity, as determined under the conventional calibration 
strategy improved by 29.65 percent (from .317 to .411). Even more impressively, under 
the normalization strategy, composite productivity values improved by 42.27 percent 
(from .317 to .451). Recalling that in this scenario, experimental assumptions include 
both learning and undersizing, it is logical to pursue investigation of alternative scenarios 
in an effort to isolate and examine the effects of these assumptions. 

The proper use of normalized effort cost data can have a significant impact on future 
software development costs. Table 14 summarizes actual project effort (MM(act)) under 
both the conventional and normalized calibration strategies. In addition, the table in- 
cludes information on potential savings which may be achieved by archiving normalized 
data in the organizational data base vice the actual cost data. These savings could result 
when, in the future, the organization is faced with estimation of a project of similar size 
and scope. By using normalized data as input, estimates would not be biased by the inef- 
ficiencies which plagued the previous project. The potential savings in our problem set 
are noteworthy, both in terms of real effort cost savings (2.2 to 34.4 man-months) and 


percentage of reduction in cost (1.05 to 16.92 percent). Figure 11 graphically represents 
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Table 14. Potential Savings Through Normalization 
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Figure 11. Potential Cost Savings Achievable Through Normalization (Man-Months) 








the potential cost savings achievable through normalization of all projects, beginning 


with project cycle two. 

These savings are possible since normalization removes the inefficiencies which lead 
to smaller COCOMO coefficients, which in turn, lead to “tighter” (i.e., smaller) cost esti- 
mates. On the other hand, the conventional calibration strategy produces higher calibra- 
tion coefficients which subsequently lead to larger size estimates (Figure 12). As 
discussed in Chapter II, these higher-than-ideal estimates significantly influence the pro- 
ject’s final results. Work expands to fill the available slack time, and the self-fulfilling 
prophecy of Parkinson's Law is realized once again (Boehm, 1981, p. 592). 

Estimated project productivity was calculated as a measure by which the effects of 
project size underestimation could be observed on project behavior and outcome. Its cal- 
culation differs from that of actual productivity in that the actual size of the project 
(KDS\(act)) is divided by the COCOMO-generated estimate of project cost based on no 
size underestimation (MM(est)). With post-facto knowledge of a project's actual size, an 
estimated project effort value can be generated for the denominator value (MM(est)). 
Figure 13 plots estimated project productivity versus project size for project cycle one 
and both the conventional and normalized estimated productivity values for project cycle 
six. It is clear from the plot that estimated productivity decreases as project size increases 
in all three instances. 

As defined, the estimated productivity value should “shadow” the actual productivity 


value as it relates to the COCOMO-calibrated project effort estimate. When compared 
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Figure 12. COCOMO Calibration Coefficient: Conventional vs Normalized - Base Case a 
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Figure 13. Estimated Productivity - Base Case 








against actual project productivity, estimated project productivity provides an indication 
of the relative accuracy and validity of the software estimation tool and its calibration co- 
efficient. Figures 14, 15, and 16 compare actual versus estimated project productivity as 
a function of project size for project cycles one and both the raw and normalized in- 
stances of project cycle six, respectively. In Figure 14, the trend toward convergence of 
the actual and estimated productivity values appears loosely related to initial project un- 
dersizing. For example, the project with the smallest size underestimation (80 KDSI with 


10% underestimation) has an actual productivity figure closest to its estimated productiv- 


ity value. Likewise, the actual productivity of the project with the largest undersizing (70 


KDSI with 50% underestimation) is furthest away from its estimated counterpart. 

From Figure 13, it is evident that the conventional COCOMO calibration method has 
lead to estimated productivity values in project cycle six approximately 10 percent more 
than similar projects in cycle one. The normalization method yields values nearly 41 per- 
cent higher than cycle one. Nevertheless, from Figure 15, conventional cycle six actual 
productivity values exceeded their estimates by between 5.1 and 14 percent. With the 
exception of the largely undersized project (70 KDSI, 50-percent undersizing), the nor- 
malization strategy, shown in Figure 16, provides the best "fit", with estimated produc- 
tivities ex.:eeding actual productivities by an average of less than 1.5 percent. 

This fact 1s also confirmed by using the completed project results for ex-post-facto 


evaluation of the accuracy of the COCOMO estimation model. The percentage of 
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. Figure 14. Actual vs Estimated Productivity: Cycle One - Conventional Strategy - Base Case 
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Figure 15. Actual vs Estimated Productivity: Cycle Six - Conventional Strategy - Base Case 
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Figure 16. Actual vs Estimated Productivity: Cycle Six - Normalization Strategy - Base Case 
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relative error in the accuracy of project cost estimation can be caluclated using the fol- 


lowing equation: 


100+|AM(act)}-MM(est)| 


Percent Relative Error = MM(act) 


(4.1) 


Equation 4.1 is used to determine the accuracy of the base case estimates generated under 
both the conventional and normalized calibration strategies in cycles two through six of 
the exercise scenario. Figure 17 is a plot of the average error for all projects by project 
cycle, and the results suggest that the accuracy of COCOMO project cost estimation in 
this scenario favors the normalized calibration mode! over the conventional model. 
C. EFFECTS OF NO UNDERSIZING ON BASE CASE RESULTS 

Having concluded an examination of conventional versus normalized calibration 


strategies in a scenario that included both learning and undersizing (base case), the pro- 


ject set was re-simulated under similar conditions, but assuming no undersizing. The 


methodology was identical to the base case, with the exception that the SD simulator in- 
put UNDEST was set at "0" in each project simulation to reflect "perfect" size estimation. 
Appendices D, E, and F document the results of these re-simulations, again modeling 
both the conventional and normalized calibration strategies. The results are summarized 
in Table 15. 

A comparison with the base case results (Table 13) reveals some interesting findings. 
With no undersizing, individual productivity improved in all projects and across all pro- 
ject cycles with respect to their undersized counterparts. In 18 of the 30 project serials, 


however, the percentage of improvement in productivity realized through the 
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Figure 17. Average Error in Accuracy of Estimation of Project Cost: Conventional vs Normalized 
Calibration Strategies - Base Case 
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Table 15. Comparison of Conventional and Normalized Calibration Strategies: 
Case With Learning and No Undersizing 


normalization process, was /ess in this scenario (no undersizing) than in the base case 
(with undersizing). This is reflected in Figure 18, where a plot of the average improve- 
ment in productivities as a result of normalization shows minimal variance between the 
two scenarios. 

Composite cycle productivities within the domain of the "no undersizing" scenario, 
again showed a significant improvement over the span of the six project cycles, with the 
conventional strategy yielding an improvement of 32.3 percent, and the normalization 


strategy 43.7 percent. These productivity improvements (without undersizing), however, 


are only marginally better than those realized in the base case (with undersizing). Figure 


19 presents a graphical summary of composite cycle productivity, comparing raw and 
normalized results in both the undersizing and no-undersizing scenario. It is evident that 
by the third project cycle, composite productivity under the normalized calibration strat- 
egy surpasses the productivity values achieved under the conventional calibration strat- 
egy, regardless of whether or not the project's size was underestimated. This finding 
suggests that normalization may be an effective tool that can help offset the negative ef- 
fects of project estimation undersizing. Nevertheless, further research is required to sup- 
port this claim. 

Estimated productivity comparisons under this scenario reveal some interesting re- 
sults. With no undersizing, actual and estimated individual project productivities are 
nearly identical in cycle one (Figure 20). These values are the same in the conventional 


and normalized cases, since the initial effect of the normalization process is not evident 
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Figure 18. Average Improvement In Productivity: Base Case vs Case With Learning and No Undersizing. 








Composite Productivity - Conventional vs Normalized Calibration 
(Base Case vs Case With Learning Factors and No Undersizing) 











° 
an 





° 
ms 
an 





Composite Productivity 
oS 
> 























0.35 
0.3 aie 
Four 
Project Cycle 
wm Raw Data - WITH Undersizing a Normalized Data - WITH Undersizing 


-& Raw Data - NO Undersizing 4 Normalized Data - NO Undersizing 








Figure 19. Composite Productivity: Conventional vs Normalized - Base Case vs Case With 
Learning and No Undersizing 


68 














Actual vs Estimated Productivity - Cycle One 
(Conventional Calibration Strategy - Case With Learning and No Undersizing) 
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until project cycle two. However, using the conventional strategy (raw data), estimates 
of productivity begin to drift, and by cycle six lag actual productivities by a range of 6.8 
percent to 10.86 percent (Figure 21). Conversely, normalized data continues to produce 
precise estimates within one percent of actual productivity values in cycle six (Figure 
22). This would indicate a more responsive calibration of the COCOMO constant by the 
normalization process in this scenario. 

The relative error in the accuracy of COCOMO's project cost estimation under con- 
ventional and ncrmalized calibration strategies is quite dramatic in this scenario of no 
undersizing, as can be clearly seen in Figure 23. With "perfect" size input, normalization 


of the data results in consistent COCOMO cost estimates across all project cycles, with a 


relative error rate of less than one-half percent. Conversely, while conventionally- 


calibrated COCOMO produces "tight" cost estimates in project cycles one and two, the 


error rate balloons to nearly ten percent by cycle six. 


D. EFFECTS OF NO LEARNING ON BASE CASE RESULTS 

In this experiment, the project set was re-simulated in a scenario which included un- 
dersizing, but assumed no learning between project cycles. The methodology differed 
from the base case only in the fact that the SD simulator parameter DSIPTK remained 
fixed at the default value of "60" for all project simulations. This effectively eliminated 
the learning assumption, by modeling the experiment with a "flat" delivered-source- 
instruction-per-task rate from cycle to cycle. Appendices G, H, and I document the results 


of the experiment. 
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Actual vs Estimated Productivity - Cycle Six 
(Conventional Calibration Strategy - Case With Learning and No Undersizing) 
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Figure 21. Actual vs Estimated Productivity: Cycle Six - Conventional Strategy - Case With Learning and 
No Undersizing 
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Figure 22. Actual vs Estimated Productivity: Cycle Six - Normalized Strategy - Case With Learning and 
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Figure 23. Average Error in Accuracy of Estimation of Project Cost: Case With Learning and No 
Undersizing 





Results of the experiment are summarized in Table 16, and show that while individ- 
ual project productivity using the conventional calibration strategy varied between .26 
and .35, composite productivity through the six project cycles decreased marginally from 
.317 to .311 (1.89 percent). In this scenario (undersizing but no learning), the normali- 
zation strategy yielded minimal improvement, at best, over the conventional strategy in 
terms of real effort (-2.92 percent to 6.54 percent), individual project productivity (-3.85 
percent to 6.25 percent) and comy site productivity (.33 percent to 3.57 percent). In ad- 
dition, with normalization, composite productivity over the six project cycles improved 
only trivially from .317 to .318 (315 percent). These composite productivity values are 
graphically represented in Figure 24, and provide an important observation. The findings 
suggest that, in an environment devoid of learning, both the conventional and normaliza- 
tion calibration strategies are largely ineffective in improving productivity. 

Similarly, both estimated productivity and relative accuracy values are inconclusive 
in this scenario. In the case of the conventional strategy, raw data values produce under- 
estimates of productivity averaging 4.5 percent, while the normalization strategy yields 
overestimates averaging 8.9 percent. The accuracy of project cost estimation favors the 
conventional COCOMO calibration strategy in three of the five project serials, besting 


the normalized model's average relative error rate, 6.08 percent to 7.62 percent. 
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Table 16. Comparison of Conventional and Normalized Calibration Strategies: 
Case With Undersizing and No Learning 
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Figure 24. Composite Productivity: Conventional vs Normalized - Case With Undersizing and No 
Learning 





E. THE EFFECTS OF OVERESTIMATION AND UNDERESTIMATION OF 
PRODUCTIVITY ON SIMULATION RESULTS 

The final series of experiments examines the impact of overestimation / underestima- 
tion of productivity on project set results. In this scenario, we again assume undersizing 
and no learning, as in the previous experiment. However, this experiment explores the 
effect of misrepresenting productivity by virtue of how a "task" is defined. 

Central to the notion of variable task definition is the situation where different soft- 
ware development organizations require different development efforts to design and code 
projects of a similar size and scope. Consequently, where DSI is constant and fixed in 
both organizations, the value of "task" becomes the determinant with regard to measuring 
effort. 

First, the project set is re-simulated with underestimation and no learning, but with a 
DSIPTK value fixed at 75 percent of the nominal case. The nominal case default value 
of the SD simulator is "60", hence, the input metric is set at "45". Cost and productivity 
values are calculated in the usual manner, using both the conventional and normalization 
calibration strategies. Data and calculations are presented in Appendices J, K, and L, and 
are Summarized in Table 17. A comparison with Table 16 values (undersizing, no learn- 
ing, nominal DSIPTK value), and employing the conventional strategy with raw histori- 
cal data, reveals significantly lower individual project productivities in each instance. 
Likewise, composite cycle productivities fall by 15.5 percent to 17.8 percent. The effects 


of normalization under these experimental conditions are negligible. Both individual 
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Table 17. Comparison of Conventional and Normalized Calibration Strategies: 


Case With Undersizing, No Learning and DSIPTK = 75% of Nominal Case 
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project productivities and composite cycle productivities are virtually unchanged despite 


normalization (improvement range of -.38 percent to 2.37 percent). 

Next, the DSIPTK value was set at 125 percent of the nominal case, or "75", and the 
projects re-simulated yet again with all other conditions unchanged. Supporting data and 
calculations are presented in Appendices M, N, and O, and are summarized in Table 18. 
Results under the conventional strategy reveal a global improvement in individual project 
productivity. Similarly, composite cycle productivity improves by an average of 10.34 | 
percent over Table 16 (nominal) values. The effect of normalization in this scenario, 
while not as dramatic as under the learning assumption (Table 13), nevertheless improves 
composite productivity by an average of 11.96 percent over the Table 16 values, and 
yields an improvement over conventional strategy values ranging from 2.09 to 4.85 
percent. 

Figure 25 is a graphical representation of composite productivity under all exercise 
conditions described in this section, and includes data carried forward from the previous 
section (DSIPTK = 100%) for comparison purposes. The composite productivity posi- 
tioning is readily apparent and appears directly linked to DSIPTK values’percentages. 
The figure also provides a view of the effects of normalization on each of the three data 


sets. Clearly, the higher DSIPTK values yield the more significant normalization benefit. 
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Table 18. Comparison of Conventional and Normalized Calibration Strategies: 
Case With Undersizing, No Learning and DSIPTK = 125% of Nominal Case 
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Figure 25. Composite Productivity: Conventional vs Normalized 
DSIPTK = 75%, 100%, and 125% 








Vv. CONCLUSIONS 


A. SUMMARY OF FINDINGS AND IMPLICATIONS 

The major objective of this thesis was to use simulation modeling to replicate the de- 
velopment of a set of 30 hypothetical software projects, the results of which were used to 
evaluate two competing calibration strategies for the COCOMO software estimation tool 


in four experimental scenarios. 
1. Phase One 
In phase one, the simulated project costs obtained by applying the conventional 


calibration strategy, were evaluated against a similar set of cost values obtained by 
applying the normalized calibration strategy in a scenario which assumed both learning 
and undersizing. The normalization process contributed to significant increases in both 
individual project productivity and composite cycle productivity. The experiment 
demonstrated that normalization provided the organization with more optimal calibration 
coefficients which, in turn, lead to more optimal cost estimations. As inefficiencies were 
eliminated in project cost estimation, simulations produced projects with lower actual 
costs, and hence, improved productivity. 

The experiment also demonstrated that the normalization strategy provided the soft- 
ware organization with the potential for significant future cost savings. The normaliza- 
tion process effectively removed many of the inefficiencies associated with undersized 


projects. Consequently, archiving normalized cost data in the organizational data base 
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vice the actual project results, produced more optimal estimates when identical projects 
were re-simulated following model calibration. In contrast, as a result of higher calibra- 


tion coefficients, the conventional calibration strategy produced consistently larger and 


less optimal cosi estizaates. 


Post-facto knowledge of the projects’ actual size was used to calculate two related ex- 
ercise metrics, both of which provided an indication of the relative accuracy and validity 
of the software estimation tool -- estimated productivity and relative error in cost estima- 
tion. The normalized cost data produced the strongest correlation between actual and es- 
timated productivity results, indicating that the model provided more accurate estimates. 
This was confirmed when the computed accuracy of the base case COCOMO estimates 


clearly favored the normalized calibration model. 
2. Phase Two 
In phase two, the base case results of phase one were compared with simulated 


results of a new case assuming learning, but no undersizing. With no undersizing, both 
the conventional and normalized calibration strategies produced global improvements in 
project productivities over base case results. Normalization again provided cost benefit 
over raw historical data, but in this scenario, the average improvement in individual 
project productivity was less dramatic than in the base case. Similarly, composite cycle 
productivities were only marginally improved over their base case counterparts. These 
findings suggest that normalization may be an effective strategy to counterbalance the 


detrimental effects of initial project undersizing. Both estimated productivity and 





relative accuracy solutions in this scenario revealed that the conventional calibration 
strategy produced increasingly suboptimal model performance over the six project cycles 
Coury: the normalized model continued to prov.ie extremely precise estimates 
throughout all project cycles. 

3. Phase Three 


Phase three re-simulated the project set in a scenario which included project size 
underestimation, but no learning. Normalization was least effective in this scenario, 
yielding minimal improvement, at best, over the conventional strategy in all key cost and 
productivity metrics. The findings suggest that without learning, both the conventional 
and normalization calibration strategies are largely ineffective in improving productivity. 
A comparison of relative model accuracy was also inconclusive in this scenario. 


4. Phase Four 


The final phase of the experiment investigated the impact of both underestimation and 
overestimation of productivity on the results of the phase three experiment. First, with 
productivity underestimated by a factor of 75 percent of the nominal case, all productiv- 
ity metrics were degraded, and normalization had a negligible impact. Next, with produc- 
tivity overestimated by a factor of 125 percent of the nominal case, all productivity 
values showed improvement. Normalization was again effective in this scenario, but less 
dramatically than in the base case (learning and no undersizing). Productivity in this sce- 


nario appears directly linked to the concept of variable task definition as it relates to the 


number of delivered source instructions per task (DSIPTK). In addition, the effects of 





normalization also tend to follow this DSIPTK movement — the higher DSIPTK values 
yield the more significant normalization benefit. 
B. FURTHER RESEARCH RECOMMENDATIONS 

Three interesting research directions could be pursued as follow-on to this study. The 
first possibility is a validation of the findings of this simulation-based study by conduct- 
ing an experiment in a real organization to compare the iwo strategies. Second, the cur- 
rent normalization strategy seeks to eliminate the inefficiencies caused by undersizing. 
The SD simulator could be used to examine the possibilities of eliminating other sources 
of inefficiency such as the misallocation of staff resources. Third, the normalization 
process requires repeated simulations to arrive at the optimal cost solution, and as such, 


is quite labor and time-intensive. The possibility for automating the process, perhaps em- 


ploying artificial intelligence techniques, could be investigated. 
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APPENDIX D. CONVENTIONAL CALIBRATION STRATEGY: 
LEARNING - NO UNDERSIZING 
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APPENDIX E. NORMALIZATION CALIBRATION STRATEGY: 
LEARNING - NO UNDERSIZING 
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APPENDIX G. CONVENTIONAL CALIBRATION STRATEGY: 
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APPENDIX H. NORMALIZATION CALIBRATION STRATEGY: 
UNDERSIZING - NO LEARNING 
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APPENDIX I. NORMALIZATION DATA: 
UNDERSIZING - NO LEARNING 
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APPENDIX J. CONVENTIONAL CALIBRATION STRATEGY: 
UNDERSIZING - 75% DSIPTK 
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5 75 80 10 72 214 192 _— 





xy 














Pra Saal -ROST oa) RAN ost) | TP sr HS ie 
Ca A 1 
40 149.1 138.1 ix 6025 
oO. 228.3 a 
80 
70 


306.7 3704 | 100 ; 37040 , 71432 pa 21501 
= 87 | 2725 95157 7569 23070 3.27 


CYCLE #3 (Raw Data. 75% OSIPTK. With Underestimation) 
ee A eh ne a) AE oe ee _T =e LMM (act) | TDEV (act)} 
P2828 


Prog. Sertal | ROST cect) | MM (oct) MiMact) O | MiMlectyrCl sum Mavkect)O, CP sum One | Coethosent [praductyay Comp Prod 
283. 1 L008 : B, . 23129 23129 Poe 59 i 0.26 


CYCLE #4 (Raw Data, 75% OSIPTK, With Underestimation) 
[Proj senat FISIPTK (%) KOS! (act) | Under (%) | KOS! (est) | MM (est) TOEV (est) | pe ae el 
SK EC (|: |: PD ee 
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CYCLE #5 (Raw Data, 75% OSIPTK. With Underestimaton) 




































































Prot Sotal REST CoML COMM C2) RMN 2 CEVRSICT CcN  T ALETEND clvay comp Prod] 
i 0 4.9 . : 1S OUD : 6994 2304 r4 0 


Ee) 1 1998 ' 178.7 61 10001 17895 | 3721 | GB 1 0.28 
_ 74 






023 
3.12.) 0.27 
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APPENDIX K. NORMALIZATION CALIBRATION STRATEGY: 
UNDERSIZING - 75% DSIPTK 


CYCLE #1 (Raw Data, 75% OSIPTK. With Underestimation) 
act BSS aS Sco SACU AEC ee Aa | ____T facdhTOEV oct) 





GERSON CCM oD a aac Cais oa UeaE -) 


1 ty) 

2 aaa 14551 1812 1745: 61 ! TOE 17231 
3 . t767 ' os ' 2125 "74 15725 32056 
4 70 2078 | W788 | 2814 87 | 21672 S628 
5 80 2302808 280 100 28000 





! fl 1 ‘ . ( 
: —:.-_-—————— 
CYCLE #2 (Normalized Data, 75% DSIPTK, With Underestimation) 
“oS OST RS ROS at) dor DE) OS ost) TT fost TEV est [ET oct} TEV (acd 
| 45 -— eee 











2 1 

1 —— ra 70 ; % a 156 

3 ry © F) eis 75 

5 75 ao | #0 OBS 16.3 
4 he} 7 DO (CBRE AE A { 





ee 1: 
1376 


370.7 


CYCLE #3 ee ee 


Fas v4 
1395 ' 150 372. ~48 
1763 | 1753 | 1742. 61 
2688) 3249 285.1 100 





CYCLE #4 (Normalized Date. 75% DSIPTK. With Underestimation) 
ae ee a ee ee } — So 








nea 


is Sana SS) To | Rey eet — STIS ASSIS — OPE sn 7 | CSAS SRA OS Pe 
Pp 4 i 











CYCLE #5 (Normalized Data. 75% DSIPTK. With Underestimation) 
| Proj Serial | cre fac | ner (Re) IS) fos NM (ood) EY (on Le a 
} 73 


a 
1212 ee es 


Raney (Care red 


CYCLE #6 (Normalized Data, 75% DSIPTK, With Underestimation) 
sat SET EROS (ot) | Urade S2) | ROS at [RM (esl) FOE V font [ee a 
75 740 16.3 -— TEs 198 
7§ 11473) 16.7 8 3s] 


35 4121.7 338 [——st tee 
i7] 259.5 


50 1769 785 a1 . . 
| 243° 278 : 7 





APPENDIX L. NORMALIZATION DATA: 
UNDERSIZING - 75% DSIPTK 


CYCLE #1, PROJECT #1 =a CYCLE #1, PROJECT #2 
a 444.7 (144 50.5 


0 6.9 


4012021 135 | 1875 
| 4020.2) 130 20.9 165 1 
ps0 {2091 1604748 | 

20. aes eins REN Nae 














\ i 
1 HN 


| 


CYCLE #1, PROJECT #4 


l 








249 ; 300 292.1 
2439 | | 265.8 
24.9 280 278.2 
249 | 270 268.9 


24.9 | 245 
24.9 240 


| 
| 
» 


CYCLE #1, PROJECT #5 











ee ee 
CYCLE #2, PROJECT #8 
LKDSI (est) | TDEV (est)|_ MM (est) | MM (act) | 


60 27.3. | 220 219.8 
60 : 205 





CYCLE #2, PROJECT #1 
KDSI (est) [TOEV (est), MM (est) | MM (act) | 
















CYCLE #2, PROJECT #5 CYCLE #2, PROJECT #4 


a. 70.7 70 228. 215 273.9 
70 22.8 265 —«264.4 
28 245 +2528 


| 285 i 
- 280 
270 





CYCLE #3, PROJECT #3 


KDSI (est) 'TDEV (est)' MM (est) | MM (act 
60 217 235.8 234.7 


| ; ; E ‘ 













CYCLE #3, PROJECT #1 CYCLE #3, PROJECT #2 


40. 204 150 . : , 1753 | 174.9 
| 470. ~-474<5 
"155 75 


CYCLE #3, PROJECT #5 
[KOSI (est) TDEV (est), MM (est) | MM (act) | 
80. 241 | 
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CYCLE #4, PROJECT #1 el CYCLE #4, PROJECT #5 


KDSi (est) TDEV (est)! MM (est) | MM(act} [  —s—s-§-§-§-_—«sxgXq KD (est) . TDEV (est)| MM (est) _ MM (act 
40 i 188 | 1424 | 144. | ti(‘(iéd 8 4.2 | 12 41.1 
40 _ 188 | 135 137.3 80 24.2 290 295.2 
40 188 | 1325 | 137.4 80 24.2 | 285 295 
40-188 =} 130 1373 [ [| 80 24.2 280 | 2946 

188 | 125 1374 [| 275 295.7 
eae 270 296.9 
bee 





188 : 120 138.3 


CYCLE #4, PROJECT #2 YCLE #4, PROJECT #3 


|KDS! (est) [TDEV (est)|_ MM (est) MM (act) | 
[203 | 194.2 | 191.1 
50. 213; 180 179.6 


| 5 ff 

"243 | ok 213 
21.3 "2. 214.2 

ir Oe 4. 185 ~~ ~216.1 

214180 —S~=—«2 16.8 


: } i 
CYCLE #5, PROJECT #4 
70 23.1 230. | — 253.6 
"23.1 =; 225 





260. |_—300 


CYCLE #5, PROJECT #3 


| KDSI (est) | TDEV (est)' MM (est) | MM (act 


182.6 
[470 «1749 
50. 202 | 165 | 1747 
50, 20.2. 155 ‘175 
5020.2 150 1768 





CYCLE #%, PROJECT #1 


[KDSI (est) | TOEV (est)! MM (est) | MM (act 
40 163° 130 371 
40 163 | 1275 | 1372 
40 | 163 =| #125 137.1 

40. 63 °° #«2+12 137.5 
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APPENDIX M. CONVENTIONAL CALIBRATION STRATEGY: 
UNDERSIZING - 125% DSIPTK 


CYCLE #1 (Raw Data, 125% DSIPTK, With Underestimation) 
eee eed CesT aE SS en ae tenn 


iO 
20 — 
3s a 
SO RE NE, 

2.5 


76.7 | 
3 207 8 20.5 bi] 
a ee ee 
[eG e se : 
i i a 
{ ; ! 


CYCLE #2 (Raw Data, 125% DSIPTK, With Underestimation) 
PPrai Seril OSIPTK CR) ROS) (act) Under (2) [ROSY (eet) | MA test) FTDEV feet) Ad act) TIDE Tae 
fo 2 | 2 | SD | © | 83.9 -——— et 1402 BT | 


1 


CYCLE #3 (Raw Data, 125% DSIPTK, With Underestimation) 
Prat Sara PSP) Ke Aaa IC = Pia ea OEY a 


175.8 

132.7 
te + +o 
a a ee a 


73.4 
174.9 
ee 
Sa ORT ea | WON oe Waco rs NSE. O75 sa OP | Catia Pincay Cop Pra 
| 4 BY | fee fT OS 





CYCLE #5 (Rew Data. 125% DOSIPTK, With Undersstimation) 
| Prot. Serial LSE Te (ey KES! (act) | Under (%) | KDS! (est) | MA jest) | JDEV fest) Tf WAM fact) “TDEV (ect)! 
je OO) BO a at 


i ae 
I A FO | 
| 104. 







____} 1889 19 

65 | 141 }——-w{ 158 i66 | 

SES RESET ENT) EN ERE ERNE OA APTA cae eG eS : 

(Pra Sera KOS! (act) | Mad (est) | Aimed) | OC)  MameclhU sum Mimectc) cre | sum t's | Costiicsent (Producthty Corp Prod] 
i 27.3 6.3; 87 17048: 
















3 60 es | 183.9 BOO 68001 S476 28521 i, _0. 
SR | 73558 T2304 0625 23% 0% 03 
eee RR Reena i . . H 


4 i 





[Pro Sena) DSIPTK (3) KDS! (act) | Under (%) | KDS! (est) | MM fest)! TREViest) fF | 
2 es 
12580 as 2 ar 
<A X  B C X 
ee WIE) 
iain 


' Ta EE ES 
Prey Sorted | REST oct) | MBN Yost) | BKecl) | O__ | MMactyU [eum Pact) Cre | sum Oe | Coothaan [Product iy Corp Prod 


a T3646 3721 ons | 0.36 

|. 3 | 6 | 1% 1688 74 | 12401 26137 | ~S476~—C;«=C«é SN , 0.36 

| 4 { 7 {| 269 B13 87 212346280 7569 19070 Y 03 
ee: Zs za} 100 Z300 | 68560 10009 [29070 236 0.36 OMS | 
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APPENDIX N. NORMALIZATION CALIBRATION STRATEGY: 
UNDERSIZING - 125% DSIPTK 








CYCLE #1 (Raw Data. 125% OSIPTK, With Underestimation) 
[ Proj Serial DSIPTK (Se KOSI (act) | Under (%) | KDSI (est) | MM (esi) [TOeViesiT FAT (act) TDEV (act) 
3 ty ts 7 z ee : P 























CYCLE #2 (Normadized Osta. 125% OSIPTK, With Underestimation) 
| Prot Serta DSIPTK (56) KOSI (act) | Under (Se) [KDS! (est) | Mi (est) TTOEV(esiT FMM fact) TOEV (act) 





a ee 70 3049 132 _| 156 |] 1993 * wi | 


Pro Serial | KOS! Tea Was Rae) a MSO] OOF a Ot roa ag Pad 
% 138.5 100.1 an 2306 6025 
a 6O: 2121 164.1 ies T % as 641 $476. 11501 


S| 80 6B 273 2086 100; 20860 “MS; 10000 ~—21507 
4, 493 | 1983 “1001 6 60161 7569 23070 2.07 
' ’ 





( ' i 1 ! 
CYCLE #3 (Normalized Data, 125% DSIPTK, With Underestimation) 
[Proj Sertal DSIPTK (%} a eee fet) Oe as AOE ec DEY (act 











i l i ! 
CYCLE #4 (Normalized Data, 125% DSIPTK, vith Underestimation) 
t Proj.Serial DSBITK { (22 RDS! fact) | Under (7) | KOS! (est) | MRd fest) | TEV (est)} aaa LM (act) | TOEV (act)| 
OF | ot eee 
SS _Y ——— 











12] 


CYCLE #5 (Normalized Data. 125% OSIPTK. With Underestimation) 


| Proj Serial | oe aS Gee STEERS SC ae ora 











[ Pro. Serial se TK (ey KUSI (act) | Under (%) | KOS! (est) | MM (est) (TOEV(eatji | | Mid (act) [TOEV (act) 
1 z pO OS CS 
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APPENDIX O. NORMALIZATION DATA: 
UNDERSIZING - 125% DSIPTK 


CYCLE #1, PROJECT #1 
PRDST Tes) [TDEY Test) WM Test)" Toc | 
98 \ 
95 


60 ; 184  : 116 153.6 


eT ie ee See 
i | ' 
CYCLE #1, PROJECT #5 


KDSI (est) |TDEV (est)| MM (est) | MM (act) 
M 


gigisiz 
Bis 
09) 


& 


88 
33/8 
nn ao 


150 | 


CYCLE #1, PROJECT #2 


KDSI (est) TDEV (est) MM (est) — MM (act) 





U 
50 
50 
50 
50 
50 
90 
90 


CYCLE #2, PROJECT #2 


KDSI (est) | TDEV (est)! MM (est) | MM (act 
30 | 189 | 140.1 139.9 
SO ___18.9 130. 129.8 














CYCLE #2, PROJECT #4 
; 60 3 oye 6 


170 





160 








20. 
20.1 
26.1 















H H 4 : 
CYCLE #8, PROJECT #4 


[KDSI (est) | TDEV (est); MM (est) | MM (act) | 


CYCLE #3, PROJECT #3 


| KDS!I (est) | TDEV (est) MM (est) — MM (act) | 








70. 191 ° 1919 , 191.3 60 197 172.5 ~~—«722 
70. | 191 | 180 1846 60 «19.7 160 —«159.7 
70 19.1 170 +1829 6019.7 ~—S«450 157.2 
7O_ | 19.1 | 160 180.4 
70 | 150 | 1799 











60 19.7120. +1536 


CYCLE #3, PROJECT #2 


40. | 191 | 9 | 100 
85 Bs 
50 ; 00 | 6139 

50 


70 i 19.4: = 145 _ 180.2 
70 | __ 19.1 ,___ 140 181 
70 19.4 130 





f 















t ' i : ‘ 
YCLE #3, PROJECT #5 CYCLE #4, PROJECT #4 


|KDSI (est) [TOEV (est) MM (est) | MM (act KDSI (est) | TDEV (est)|_ MM (est) MM (act 


| 70. ; 208 | 165 . 181.1 
70208 ~~ ~—«160 180.3 





20.8 155 
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CYCLE #4, PROJECT #1 eee CYCLE #4, PROJECT # 


IKDS! (est) [TOEV (est)) MM (est) MM(act) TL KDSI (est) TDEV (est) MM (est) MM (act 
40 . 17.6 » 104 ‘ Bi 9.4 














































05. ee a: 
40 176 100 ; 80.212 200 212 
40..~—Ss«17<6 90 100.2 
40. -~—sO«47-6 85 99.2 
40: «+176 3, «2 | | 
40 17.6 80 992 |] | 
eae 


i ‘ : 








ui L 


CYCLE #4, PROJECT #2 CYCLE #4, PROJECT #3 


DSI (est) TDEV (est est) _| MM (act KDSI (est) | TDEV (est), MM (est) | MM (act' 














60 i 189 154.8 156.5 
60 189 140 153.9 
60 18.9 135 153.1 
60 : 18.9 130 382.4 
60 ' 189 (128 152.1 
60 18.9 125 152.1 


i 18.9 120 153.2 




















CYCLE #5,PROJECT#5 = || CYCLE #5, PROJECT #4 


KOS! (est) TDEV (est) MM (est MM(act) |__| KDSt (est) TOEV (est)|_ MM (est) _ MM (act | 
1 OR XA A 





t ‘ 
Hl 


CYCLE #5, PROJECT #2 


KDSI (est) TDEV (est)' MM (est) | MM (act 
8: 1346 








CYCLE #5, PROJECT #3 


KDSI (est) _TDEV (est)| MM (est): MM (act 
BA 82 




















NO "ST 7 
! i 128.8 60 20.1 : 460 ‘169.5 
; 60 20.1 _ 140 / 163.6 
17.8 i 105 _ 135 i 


178 =: ~— 100 
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CYCLE #5, PROJECT #1 
| KDS!} (est) | TDEV (est): MM (est) — MM (act) } 
0 } 16.3 + 102.€ : 02.4 


16.3 775 
16.3 7 
O..8F=FCO630 725 
Oo. ©6633 2 
16.3 
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